Image processing method, image processing apparatus, program and recording medium

ABSTRACT

An image processing apparatus is provided. The image processing apparatus converts moving image data including plural first image data of a first resolution into moving image data including plural second image data of a second resolution. The apparatus generates position alignment data between the target first image data and another first image data other than the target first image data; aligns candidate image data having the second resolution corresponding to the target first image data with the other first image data according to the position alignment data generated; updates the candidate image data to minimize a first difference between the other first image data and the candidate image data position-aligned; and executes the aligning and the updating at least one time, and using the candidate image data having the difference equal to or smaller than a reference value, as the second image data corresponding to the target first image data.

The present application claims priority to Japanese Patent ApplicationJP 2005-237926 filed in the Japanese Patent Office on Aug. 18, 2005, theentire contents of which being incorporated herein by reference.

BACKGROUND

The present application relates to an image processing method, an imageprocessing apparatus, a program and a recording medium, respectively formaking moving image data having a high resolution.

In the field of image processing, one main idea is to improve imageresolution. Technologies for creating a high resolution image byenlarging one low resolution image such as through pixel interpolationhave been studied. There is a limit, however, in creating a high orsuper resolution image from a low resolution image because the frequencyband of a signal of the low resolution image is limited.

High resolution or super resolution technologies are well known by whicha super resolution image having a large number of pixels is created froma plurality of low resolution images. A variety of methods for superresolution analysis have been proposed. Well known methods include amethod using a frequency space (refer to “Multiframe image restorationand registration”, by R. Y. Tsai and T. S. Huang, Advances in ComputerVision and Image Processing, vol. 1, JAIPress Inc., 1984), a methodbased on MAP estimation (refer to “Extraction of high-resolution framesfrom video sequences”, by R. R. Shultz and R. L. Stevenson, IEEEtransactions on Image Processing, Vol. 5, No. 6, June 1996), a method byProjection on Convex Sets (refer to “High-resolution imagereconstruction from lower-resolution image sequences and space-varyingimage restoration”, by A. M. Tekalp, M. K. Ozkan and M. I. Sezan,Proceedings of IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), San Francisco, Calif., vol. 3, March 1992),and the like.

Although the above-described methods in related art, particularly themethod based on MAP estimation and the like, can obtain very goodresults, these methods require a number of input images, resulting in anissue of a very high calculation cost. There is another issue thatmotion compensation for object blurring is difficult.

Many super resolution technologies assume that a plurality of input lowresolution images are obtained by photographing the same scene throughstepwise parallel displacement of a camera. There arises, therefore, anissue of object blurring if each individual object moves in a scene, ifa camera is moved in a scene having a depth, or in other cases.

Although the technology by R. R. Shultz, et al deals with objectblurring, it is necessary to estimate a motion vector of an input imageat each pixel position, resulting in an issue of a high calculationcost. The issue associated with this technology is that even if a motionvector is detected from a low resolution input image, a motion vectorcannot be estimated correctly because of noises, aliasing and the like.

Accordingly, it is desirable to provide an image processing method, animage processing apparatus, a program or a recording medium,respectively being capable of generating super resolution moving imagedata from low resolution moving image data with a calculation amountsmaller than that in the related art. The present invention has beenmade in view of the issues existing in the above-described related art.

SUMMARY

According to a first aspect, there is provided an image processingmethod of converting moving image data formed with a plurality of firstimage data having a first resolution into moving image data formed witha plurality of second image data having a second resolution, theplurality of first image data being sequentially processed for display.The image processing method includes: a first step of generatingposition alignment data or positioning data between the first image datato be processed and another first image data other than the first imagedata to be processed; a second step of aligning a position of orpositioning candidate image data having the second resolutioncorresponding to the first image data to be processed with a position ofthe other first image data, in accordance with the position alignmentdata or the positioning data generated by the first step; a third stepof updating the candidate image data so as to make small a firstdifference between the other first image data and the candidate imagedata position-aligned by the second step; and a fourth step of executingthe second step and the third step one time or a plurality of times, andusing the candidate image data having the difference equal to or smallerthan a reference value, as the second image data corresponding to thefirst image data to be processed.

A second aspect provides an image processing apparatus for convertingmoving image data formed with a plurality of first image data having afirst resolution into moving image data formed with a plurality ofsecond image data having a second resolution, the plurality of firstimage data being sequentially processed for display. The imageprocessing apparatus includes: first means for generating positionalignment data between the first image data to be processed and anotherfirst image data other than the first image data to be processed; secondmeans for aligning a position of candidate image data having the secondresolution corresponding to the first image data to be processed with aposition of the other first image data, in accordance with the positionalignment data generated by the first means; third means for updatingthe candidate image data so as to make small a first difference betweenthe other first image data and the candidate image data position-alignedby the second means; and fourth means for executing processes by thesecond means and the third means one time or a plurality of times, andusing the candidate image data having the difference equal to or smallerthan a reference value, as the second image data corresponding to thefirst image data to be processed.

A third aspect provides a program to be executed by an image processingapparatus for converting moving image data formed with a plurality offirst image data having a first resolution into moving image data formedwith a plurality of second image data having a second resolution, theplurality of first image data being sequentially processed for display.The program includes: a first procedure of generating position alignmentdata between the first image data to be processed and another firstimage data other than the first image data to be processed; a secondprocedure of aligning a position of candidate image data having thesecond resolution corresponding to the first image data to be processedwith a position of the other first image data, in accordance with theposition alignment data generated by the first procedure; a thirdprocedure of updating the candidate image data so as to make small afirst difference between the other first image data and the candidateimage data position-aligned by the second procedure; and a fourthprocedure of executing the second procedure and the third procedure onetime or a plurality of times, and using the candidate image data havingthe difference equal to or smaller than a reference value, as the secondimage data corresponding to the first image data to be processed.

A fourth aspect provides a recording medium storing a program to beexecuted by an image processing apparatus for converting moving imagedata formed with a plurality of first image data having a firstresolution into moving image data formed with a plurality of secondimage data having a second resolution, the plurality of first image databeing sequentially processed for display. The program includes: a firstprocedure of generating position alignment data between the first imagedata to be processed and another first image data other than the firstimage data to be processed; a second procedure of aligning a position ofcandidate image data having the second resolution corresponding to thefirst image data to be processed with a position of the other firstimage data, in accordance with the position alignment data generated bythe first procedure; a third procedure of updating the candidate imagedata so as to make small a first difference between the other firstimage data and the candidate image data position-aligned by the secondprocedure; and a fourth procedure of executing the second procedure andthe third procedure one time or a plurality of times, and using thecandidate image data having the difference equal to or smaller than areference value, as the second image data corresponding to the firstimage data to be processed.

The present application can provide an image processing method, an imageprocessing apparatus, a program and a recording medium, respectivelybeing capable of generating super resolution moving image data from lowresolution moving image data at a calculation amount smaller than thatof the related art.

Additional features and advantages are described herein, and will beapparent from, the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a diagram showing the hardware structure of an imageprocessing apparatus according to an embodiment;

FIG. 2 is a conceptual diagram illustrating the whole procedure of asuper resolution conversion process to be executed by the imageprocessing apparatus shown in FIG. 1;

FIG. 3 is a conceptual diagram illustrating a process of generatingmoving image data MI_(k−1;)

FIG. 4 is a conceptual diagram illustrating a process of generatingmoving image data MI_(k);

FIG. 5 is a conceptual diagram illustrating a process of generating aninitial value of super resolution image data;

FIG. 6 is a conceptual diagram illustrating a process corresponding to aformula (14) according to an embodiment;

FIG. 7 is a flow chart illustrating processes to be executed by theimage processing apparatus shown in FIG. 1; and

FIG. 8 is a flow chart following the flow chart of FIG. 7, illustratingprocesses to be executed by the image processing apparatus shown in FIG.1.

DETAILED DESCRIPTION

An image processing apparatus according to an embodiment will bedescribed below. First, a correspondence will be described betweenconstituent elements of embodiments and constituent elements of theembodiment.

Low resolution image data L shown in FIG. 2 and other Figures is oneexample of first image data of the embodiment, super resolution imagedata z is one example of second image data of the embodiment.

Processing shown in FIGS. 3 and 4 and a process at Step ST1 shown inFIG. 7 are examples of a second step, a first means and a firstprocedure of the embodiment.

TR_(k−1) and TR_(k+1) shown in FIG. 6 and Step ST7 shown in FIG. 7 areexamples of a second step, a second means and a second procedure of theembodiment.

A whole procedure flow shown in FIG. 6 and Steps ST8 to ST17 shown inFIGS. 7 and 8 are examples of a third step, a fourth step, a thirdmeans, a fourth means, a third procedure and a fourth procedure of theembodiment.

FIG. 1 is a diagram showing the whole structure of an image processingapparatus 1 according to an embodiment. As shown in FIG. 1, the imageprocessing apparatus 1 has, for example, an interface 4, a memory 6 anda processing circuit 8 which are interconnected by a data line 3. Forexample, the interface 4 receives low resolution image data L shown inFIG. 2 and other Figures from an external device disposed outside theimage processing apparatus 1.

The low resolution image data L is written, for example, in the memory6. The memory 6 stores programs PRG defining operations of theprocessing circuit 8 and data to be used for processing by theprocessing circuit 8. The processing circuit 8 executes the followingsuper resolution conversion processing in accordance with the programPRG stored in the memory 6.

Prior to describing the details of the image processing apparatus 1,description will be made on the theory on which processing by the imageprocessing apparatus 1 is based. In this embodiment, description will bemade on a case which adopts a super resolution theory by MAP estimation.

First, a width of a resolution of predetermined moving image data isrepresented by w and a height thereof is represented by h. In order tosimplify the description, the moving image data is assumed to be greyscale image data. L_(k) represents image data of a frame number k amongmoving image data and an wh-dimension vector with respective pixelsbeing disposed in a scan line order. The final target is to estimate animage data sequence of a super resolution corresponding to image data ofrespective frames of moving image data.

In this embodiment, n represents an integer of 2 or larger, a resolutionof super resolution image data to be estimated has a width nw and aheight nh. Image data having a frame number k in the super resolutionimage data sequence to be estimated is represented by z_(k) which is an²wh-dimension vector with respective pixels of image data beingdisposed in a scan line order.

Assuming that super resolution image data z_(k) of a frame number kexists already, low resolution image data L₁ of a frame number 1different from k can be expressed by the following formula (1).L₁=W_(k1)z_(k)+n_(k)  (1)

W_(k1) is a matrix of wh rows and n²wh columns. Times k and 1 aredifferent. W_(k1) is a matrix for performing a motion compensationbetween times k and 1 and a conversion of generating low resolutionimage data from super resolution image data. Solving W_(k1) will belater described. Description is made by assuming that W_(k1) is knownalready. n_(k) is a wh-dimension vector representing errors to begenerated during motion compensation and resolution conversion, noiseson image data changing with time, and the like.

In this embodiment, a super resolution image data sequence {z_(k)} isestimated from low resolution input image sequence {L_(k)}. In thisembodiment, super resolution image data is estimated one frame afterframe starting from the time 0. In the following, description will bemade on estimating super resolution image data z_(k) at time k.

In general super resolution technologies, in order to estimate z_(k), aplurality set of low resolution image data is input including lowresolution image data L_(k) at time k, image data of several framesbefore and after time k. In the present invention, a image dataL1(k−a≦1≦k+a) before and after time k is input.

In this embodiment, in addition to low resolution image data, resultantimage data z_(k−1) before one frame is also used as input image data.

The resultant image data z_(k−1) before one frame has a superresolution, reflecting image data of several preceding frames. It isconsidered that a better result of z_(k) estimation can be obtained byusing the resultant image data as input data.

Super resolution technologies of related art require image data of alarge number of frames as the input low resolution image data, resultingin a very high calculation cost. In contrast, since the results of pastframes are reused in this embodiment, the amount of input low resolutionimage data can be made small and a calculation cost can be reduced.

Given low resolution image data L₁(k−a≦1≦k+a) of 2a+1 frames andresultant image data z_(k−1) of a super resolution one frame before,z_(k) is estimated. This method will be described by using maximumposteriori probability estimation (MAP estimation). The followingformula means maximizing a probability of whether an estimated value ofsuper resolution image data of a present frame k is proper, relative tolow resolution image data presently obtained as observation results andthe calculation results of preceding frames. $\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\max}{\Pr\left( {{z_{n,k}\text{❘}z_{k - 1}},\left\{ {L_{1}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}} \right)}}} & (2)\end{matrix}$

In the formula (2), z_(n, k) represent super resolution image data of aframe number k. Pr(z_(n, k)|Z_(k−1), {L1(k−a≦1≦k+a)}) indicates aconditional probability representative of whether Z_(n, k) are properwhen z_(k−1) and {L₁(k−a≦1≦k+a)} are given. Therefore, z_(n, k) giving amaximum conditional probability value are estimated to obtain superresolution image data.

The formula (2) can be modified to the following formula (3) by theBayesian rule. $\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\quad\max}\frac{{\Pr\left( {z_{k - 1},{\left\{ {L_{l}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}\text{❘}z_{n,k}}} \right)}{\Pr\left( z_{n,k} \right)}}{\Pr\left( {z_{k - 1},\left\{ {L_{l}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}} \right)}}} & (3)\end{matrix}$

Since the content of “argmax” of the formula (3) is a probability, theformula (3) can be modified to the formula (4) by using logarithm.$\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\quad\max\quad\log}\frac{{\Pr\left( {z_{k - 1},{\left\{ {L_{l}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}\text{❘}z_{n,k}}} \right)}{\Pr\left( z_{n,k} \right)}}{\Pr\left( {z_{k - 1},\left\{ {L_{l}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}} \right)}}} & (4)\end{matrix}$

Since it is not necessary to consider a probability that the contents of“argmax” of the formula (4) are not related to Z_(n. k), it is possibleto neglect the denominator component in the formula (4). Therefore, theformula (4) can be modified to the following formula (5).$\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\quad\max}\left( {{\log\quad{\Pr\left( {z_{k - 1},{\left\{ {L_{l}\left( {{k - a} \leq l \leq {k + a}} \right)} \right\}\text{❘}z_{n,k}}} \right)}} + {\log\quad{\Pr\left( z_{{n,k})} \right.}}} \right.}} & (5)\end{matrix}$

The formula (5) can further be decomposed to the following formula$\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\quad\max}\left( {{\log\quad{\Pr\left( {z_{k - 1}\text{❘}z_{n,k}} \right)}} + {\sum\limits_{l = {k - a}}^{k + a}{\log\quad{\Pr\left( {L_{l}\text{❘}z_{n,k}} \right)}}} + {\log\quad{\Pr\left( z_{n,k} \right)}}} \right)}} & (6)\end{matrix}$

Description will be made on each term in “argmax” in the right side ofthe formula (6).

First, consider “Pr(z_(k−1)|Z_(n, k))”. This indicates a probabilityrepresentative of whether the estimation result z_(k−1) of a framenumber k−1 one frame before is proper relative to the estimation valuesz_(n, k) of super resolution image data of a frame number k.

Whether z_(k−1) is proper or not can be decided from Z_(n, k) subjectedto motion compensation and an error from z_(k−1). In this embodiment,this motion compensation is represented by a matrix W_(zk). The matrixW_(zk) is a matrix of n²wh×n²wh.

For example, motion vector of Z_(n, k) at each pixel, affinetransformation of whole image data of Z_(n, k,) and the like may be usedas the matrix W_(zk). In an actual moving image data, each individualobject may move in addition to a motion of a camera. If each individualobject moves, the object may disappear in one of adjacent frames andappear in the other. In this case, even motion compensation isperformed, an error occurs in this area. This embodiment considers thiscase and defines “Pr(z_(k−1)|z_(n, k))” by the following formula (7).$\begin{matrix}{{\Pr\left( {z_{k - 1}\text{❘}z_{n,k}} \right)} = {\frac{1}{\lambda_{z}}{\exp\left( {- {{M_{zk}\left( {z_{k - 1} - {W_{zk}z_{n,k}}} \right)}}^{2}} \right)}}} & (7)\end{matrix}$

The formula (7) assumes that a probability has the Gaussiandistribution. In the formula (7), λz is a normalization constant, andM_(zk) is a matrix of n²wh×n²wh for removing an error which cannot becovered by the motion compensation W_(zk). For example, if motioncompensation for only parallel displacement components of a camera isused as W_(zk), M_(zk) detects a motion of an individual object afterW_(zk) is adopted, and if there is a motion, a gain is set to makesmaller an error absolute value after conversion, whereas if there is nomotion, the gain is set not to change the value after conversion.

Next, similarly “Pr(L₁|z_(n, k))” is defined by the following formula(8). $\begin{matrix}{{\Pr\left( {L_{l}\text{❘}z_{n,k}} \right)} = {\frac{1}{\lambda_{y}}{\exp\left( {- {{M_{kl}\left( {L_{l} - {W_{kl}z_{n,k}}} \right)}}^{2}} \right)}}} & (8)\end{matrix}$

In the formula (8), λy is a normalization constant. A matrix W_(k1)represents a coupling between motion compensation from Z_(n, k) to ascene represented by low resolution image data L_(l) and conversion fromsuper resolution image data into low resolution image data.

In accordance with the motion compensation, an area of vertical n pixelsand horizontal n pixels of super resolution image data is related toeach pixel of low resolution image data, and an average value of pixelscontained in each area is converted into a pixel value of low resolutionimage data.

The matrix W_(k1) is a matrix of wh×n²wh. M_(k1) is a matrix of wh xn²wh for removing an error which cannot be covered by W_(k1). Forexample, if a coupling between motion compensation for only paralleldisplacement components of a camera and conversion into low resolutionimage data is used as W_(k1), M_(k1) detects a motion of each individualobject after using W_(k1), and if there is a motion, a gain is set tomake smaller an error absolute value after conversion, whereas if thereis no motion, the gain is set not to change the value after conversion.

Next, “Pr(z_(n, k))” will be described.

“Pr(z_(n, k))” is a probability representative of whether the estimationvalues Z_(n, k) of super resolution image data are proper. Various termsare adopted in general super resolution technologies. For example, ifthe estimation values Z_(n, k) of super resolution image data areproper, a probability distribution assumes that Z_(n, k) are piece-wisesmooth. Namely, a probability function is taken into consideration whichtakes a high value if there are many piece-wise smooth areas and a lowvalue if there are many edged areas.

In this embodiment, “Pr(z_(n, k))” is expressed by the following formula(9). $\begin{matrix}{{\Pr\left( z_{n,k} \right)} = {\frac{1}{\lambda_{C}}{\exp\left( {- {{Cz}_{n,k}}^{2}} \right)}}} & (9)\end{matrix}$

In the formula (9), λc is a normalization constant. C in the formula (9)is a matrix of n²wh×n²wh which is a high pass filter. An output of thehigh pass filter takes a small value for a smooth area and a large valuefor an edged area. Therefore, in the formula (9), if there are many fineedges in Z_(n, k), then “Pr(z_(n, k))” takes a small value, whereasthere are no fine edge, “Pr(z_(n, k))” takes a large value.

The formula (6) is transformed to the following formula (10) byconsidering the formulas (7), (8) and (9) and that λ_(z), λ_(y) andλ_(C) are constants. $\begin{matrix}\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{\arg\quad\max}\begin{pmatrix}{\log\left( {{\frac{1}{\lambda_{z}}{\exp\left( {- {{M_{zk}\left( {z_{k1} - {W_{zk}z_{n,k}}} \right)}}^{2}} \right)}} +} \right.} \\{{\sum\limits_{l = {k - a}}^{k + a}{\log\left( {\frac{1}{\lambda_{y}}{\exp\left( {- {{M_{kl}\left( {L_{l} - {W_{kl}z_{n,k}}} \right)}}^{2}} \right)}} \right)}} +} \\{\log\left( {\frac{1}{\lambda_{C}}{\exp\left( {- {{Cz}_{n,k}}^{2}} \right)}} \right)}\end{pmatrix}}} \\{= {\underset{z_{k}^{\prime}}{\arg\quad\max}\begin{pmatrix}{{- {{M_{zk}\left( {z_{k - 1} - {W_{zk}z_{n,k}}} \right)}}^{2}} -} \\{{\sum\limits_{l - k - a}^{k - a}{{M_{kl}\left( {L_{l} - {W_{kl}z_{n,k}}} \right)}}^{2}} - {{Cz}_{n,k}}^{2}}\end{pmatrix}}} \\{= {\underset{z_{k}^{\prime}}{\arg\quad\max}\begin{pmatrix}{{\quad{M_{\quad{zk}}\left( \quad{z_{\quad{k\quad 1}} - {W_{zk}\quad z_{\quad{n,\quad k}}}} \right)}}^{2} +} \\{{\sum\limits_{l = {k - a}}^{\quad{k\quad + \quad a}}{{M_{\quad{kl}}\left( {L_{\quad l} - {W_{\quad{kl}}z_{\quad{nk}}}} \right)}}^{2}} +} \\{{Cz}_{n,k}}^{2}\end{pmatrix}}}\end{matrix} & (10)\end{matrix}$

A function f is defined by the following formula (11). $\begin{matrix}{{f\left( z_{n,k} \right)} = {{{M_{zk}\left( {z_{k - 1} - {W_{zk}z_{n,k}}} \right)}}^{2} + {\sum\limits_{l = {k - a}}^{k + a}{{M_{kl}\left( {L_{l} - {W_{kl}z_{n,k}}} \right)}}^{2}} + {{Cz}_{n,k}}^{2}}} & (11)\end{matrix}$

By using the function f shown in the formula (11), the formula (10) istransformed to the following formula (12). $\begin{matrix}{z_{k} = {\underset{z_{k}^{\prime}}{{argmin}\quad}{f\left( z_{n,k} \right)}}} & (12)\end{matrix}$

The issue described herein is an ill-posed inverse problem, so that itcannot be the that a sole solution exists. Therefore, in order to solvethe minimizing problem of the formula (12), an optimization method suchas the steepest descent method is used.

The steepest descent method is realized by repeating the processingrepresented by the following formula (13).z _(n+1.k) =z _(n,k) −αΔf(z _(n,k))  (13)

In the formula (13), Z_(n, k) are estimation values of super resolutionimage data to be obtained by repeating the processing n times.

A gradient Δf of the function f is given by the following formula (14)based on the formula (11). $\begin{matrix}{{\Delta\quad{f\left( z_{n,k} \right)}} = {{2W_{Z}^{T}{M_{z}^{T} \cdot {M_{z}\left( {{W_{z}z_{n,k}} - z_{k - 1}} \right)}}} + {2{\sum\limits_{l = {k - a}}^{k + a}{W_{l}^{T}{M_{l}^{T} \cdot {M_{l}\left( {{W_{l}z_{n,k}} - L_{l}} \right)}}}}} + {2{Cz}_{n,k}}}} & (14)\end{matrix}$

In this embodiment, the processing circuit 8 shown in FIG. 1 repeats theprocessing given by the formula (13). In this case, low resolution imagedata L_(k) subjected to an enlargement processing is used as initialestimation values z0 and k of super resolution image data.

Bicubic interpolation generally used or the like is used as theenlargement processing. When the first frame of a moving image datasequence is to be processed, resultant image data z_(k−1) one framebefore does not exist. Therefore, the term corresponding to this in theformula (14) is not calculated.

Further, when the moving image data sequence of first a frames and lasta frames are converted into super resolution images, there are somemissing frames of input low resolution image data L₁ in the formula(14). In this case, the processing circuit 8 calculates the formula (14)by using only effective input low resolution image data L₁.

The processing circuit 8 of the image processing apparatus 1 shown inFIG. 1 repetitively calculates the formulas (13) and (14) untilpredetermined conditions are satisfied, in accordance with the programPRG stored in the memory 6 to thereby obtain super resolution image dataZ_(k). The processing by the processing circuit 8 of the embodiment isexecuted in accordance with the program PRG.

In this embodiment, the processing by the formulas (13) and (14) arecalled a super resolution conversion processing. In the following,description will be made on an algorithm for executing the processing ofthe formulas (13) and (14).

The processing circuit 8 shown in FIG. 1 generates high resolutionmoving image data HMP by subjecting low resolution moving image data LMPshown in FIG. 2 to the super resolution conversion processing.

Specifically, the processing circuit 8 generates super resolution imagedata Z by sequentially executing the super resolution conversionprocessing for each of a plurality of low resolution image data Lconstituting the moving image data LMP, for example, in the displayorder.

In this case, for example, by using low resolution image data of aprocessing object, low resolution image data before and after the objectlow resolution image data, super resolution image data corresponding tolow resolution image data before one display order of the object lowresolution image data, and motion image data, the processing circuit 8generates super resolution image data corresponding to the object lowresolution image data.

In the example shown in FIG. 2, by using low resolution image data L_(k)of a processing object, low resolution image data L_(k−1), and L_(k+1),before and after the object low resolution image data, super resolutionimage data Z_(k−1) corresponding to low resolution image data L_(k−1)before one display order of the object low resolution image data L_(k),and motion image data MI_(k−1), the processing circuit 8 generates superresolution image data Z_(k) corresponding to the object low resolutionimage data L_(k) by executing the super resolution conversion processingSP_(k).

The super resolution image data Z_(k−1) and motion image data MI_(k−1)are generated by the super resolution conversion processing SP_(k−1) andstored in the memory 6.

Generation of Moving Image Data MI_(k−1)

The processing circuit 8 generates motion image data MI_(k−1) byexecuting the processing shown in FIG. 3.

Namely as shown in FIG. 3, the processing circuit 8 performs paralleldisplacement of the low resolution image data L_(k−1) using the lowresolution image data L_(k) as a reference of position alignment togenerate low resolution image data La_(k−1) (parallel displacementprocessing PM_(k−1)).

In the position alignment, the processing circuit 8 generates paralleldisplacement (i, j) by using the following formula (15), where (i, j) isparallel displacement, L_(k−1), (s-i, t-j) is pixel data of the lowresolution image data L_(k−1) at a pixel position (s-i, t-j), and L_(k)(s, t) is pixel data of the low resolution image data La_(k−1). Namely,the processing circuit 8 determines the parallel displacement data (i,j) in such as manner that an absolute value sum of differences of pixelsof the low resolution image data La_(k−1) after the paralleldisplacement and the low resolution image data L_(k) becomes minimum.$\begin{matrix}{\left( {i,j} \right) = {\underset{i,j}{argmin}{\sum\limits_{s,t}{{{L_{k}\left( {{s - i},{t - j}} \right)} - {L_{k - 1}\left( {s,t} \right)}}}}}} & (15)\end{matrix}$

If an enlargement factor of the super resolution conversion processingof the embodiment is n, it is necessary to calculate (i, j) in the unitof 1/n pixel. The image position alignment method includes a blockmatching method of calculating a total sum of difference absolute valuesof pixels of all combinations of (i, j), a Lucas-Kanade method using aniterative process (“An Iterative Image Registration Technique with anApplication to Stereo Vision”, by B. D. Lucas and T. kanade, Proceedingsof Imaging Understanding Workshop, 1981), and the like. Any one of thesemethods may be used in this embodiment.

Next, as shown in FIG. 3, the processing circuit 8 generates differenceabsolute value data representative of the difference absolute value inthe pixel unit between the low resolution image data La_(k−1) generatedby the parallel displacement processing PM_(k−1) and the low resolutionimage data L_(k) (difference absolute value calculation DC_(k−1)).

The difference absolute value data is obtained by detecting a motion ofeach individual object other the motion of the whole image. Eachindividual object with motion takes a large pixel value, whereas eachindividual object without motion takes a small pixel value.

The pixel value of the difference absolute value data is influenced notonly by the motion itself of an object but also by noises, aliasing,position alignment errors and the like. It is herein assumed that thepixel value of image data to be generated by factors other than a motionof an object contains a large amount of relatively high frequencycomponents. Namely it is assumed that object blurring changes a pixelvalue in a relatively large range, whereas noises and position alignmenterrors change a pixel value in a very local area.

The processing circuit 8 subjects the difference absolute value data tothe processing of removing high frequency components by using a low passfilter, a median value filter or the like in a post-processing AP_(k−1),to thereby extract only a motion of an object and generate the movingimage data MI_(k−1).

Generation of Moving Image Data MIk

As shown in FIG. 4, the processing circuit 8 generates positionalignment data PR_(k) and moving image data MI_(k) by executing theprocessing similar to those described with reference to FIG. 3 for thelow resolution image data L_(k) and L_(k+1).

Generation of Z_(0, k)

The processing circuit 8 generates the super resolution estimationinitial image data Z_(0, k) of the super resolution image data Z_(n, k)shown in the formulas (13) and (14), in the manner illustrated in FIG.5.

The processing circuit 8 executes an enlargement processing to convertthe low resolution image data L_(k) to have n pixels in vertical andhorizontal directions.

The enlargement processing is performed by a bicubic interpolationmethod or the like which are used generally and frequently.

Next, by using the position alignment data PR_(k−1) representative ofthe parallel displacement data (i, j) generated by the paralleldisplacement processing PM_(k−l) shown in FIG. 3, the processing circuit8 performs parallel displacement of the super resolution image dataZ_(k−1) generated by the super resolution conversion processing SP_(k−1)shown in FIG. 2 to align the position of the super resolution image datawith the position of the low resolution image data L_(k) after theenlargement processing. This aligned super resolution image data is usedas the super resolution estimation initial image data Z_(0, k).

Update Processing of Z_(n,k)

FIG. 6 is a diagram illustrating the operation of the processing circuit8 which sequentially updates the super resolution image data Z_(n, k) byusing the formulas (13) and (14). FIG. 6 illustrates the operation underthe condition of “a=1” in the formula (14).

The processing circuit 8 calculates the second term of the formula (14)by executing processing EX_(k−1), EX_(k) and EX_(k+1) shown in FIG. 6.

The processing circuit 8 calculates the first term of the formula (14)by executing a position shift processing TR1, a difference processingSUB1 and a gradient image update processing D1 shown in FIG. 6.

The processing circuit 8 calculates the third term of the formula (14)by executing an image update processing D2 shown in FIG. 6.

First, the processing EX_(k−1) shown in FIG. 6 will be described.

The processing EX_(k−1) is executed for the second term 1=k−1 of theformula (14) and includes a position shift/resolution conversionprocessing TR_(k−1), a difference processing SUB_(k−1) and a gradientimage generation processing D_(k−1).

In the position shift/resolution conversion processing TR_(k−1), byusing the position alignment data PR_(k−1) generated by the paralleldisplacement processing PM_(k−1) shown in FIG. 3, the processing circuit8 aligns the position of the super resolution estimation image dataZ_(n, k) with the position of the low resolution image data L_(k−1) andthereafter converts the resolution into a low resolution. The positionshift/resolution conversion processing TR_(k−1) is indicated by thematrix W_(k−1) in the formula (14).

Next, in the difference processing SUB_(k−1), the processing circuit 8generates difference image data “W_(k−1)Z_(n, k)−K_(k−1)” representativeof a difference between the low resolution image data generated by theposition shift/resolution conversion processing TR_(k−1) and the lowresolution image data L_(k−1).

Next, in the gradient image generation processing D_(k−1), theprocessing circuit generates the matrices M_(k−1) and M_(k−1) in theformula (14) by using the moving image data MI_(k−1) generated by theprocessing shown in FIG. 3.

The matrix M_(k−1) does not change the value of an object withoutmotion, but lowers the value of an object with motion. Since the lowresolution image data L_(k−1) is used as a reference pixel position inthe matrix M1, the pixel position of the moving image data MI_(k−1) iscorrected.

In the gradient image generation processing D_(k−1), the processingcircuit 8 converts the difference image data generated by the differenceprocessing SUB_(k−1) by using the matrix W_(k−1) ^(T)M_(k−1) ^(T)M_(k−1)to generate “W_(k−1)M_(k−1) ^(T)M_(k−1)(W_(k−1)Z_(n, k)-L_(k−1))”.

Next, the processing EX_(k) shown in FIG. 6 will be described.

The processing EX_(k) is executed for the second term 1=k of the formula(14) and includes a position shift/resolution conversion processingTR_(k), a difference processing SUB_(k) and a gradient imagegeneration/update processing D_(k).

In the position shift/resolution conversion processing TR_(k), theprocessing circuit 8 converts the super resolution estimation image dataZ_(n, k) into the low resolution image data.

The position shift/resolution conversion processing TR_(k) is indicatedby the matrix W_(k) in the formula (14).

Next, in the difference processing SUB_(k), the processing circuit 8generates difference image data “W_(k)Z_(n, k)−L_(k)” representative ofa difference between the low resolution estimation image data generatedby the position shift/resolution conversion processing TR_(k) and thelow resolution image data L_(k).

Next, in the gradient image generation/update processing D_(k), theprocessing circuit 8 does not execute the motion compensation processingbut generates “W_(k) ^(T)(W_(k)Z_(n, k)−L_(k))” by multiplying only thematrix W_(k) ^(T).

In the gradient image generation/update processing D_(k), the processingcircuit 8 adds “W_(k−1)M_(k−1) ^(T)M_(k−1) (W_(k−1)Z_(n, k)−L_(k−1))”generated by the gradient image generation processing D_(k−1) and “W_(k)^(T)(W_(k)Z_(n, k)−L_(k))”.

Next, the processing EX_(k+1) shown in FIG. 6 will be described.

The processing EX_(k+1) is executed for the second term 1=k+1 in theformula (14) and includes a position shift/resolution conversionprocessing TR_(k+1), a difference processing SUB_(k+1) and a gradientimage generation/update processing D_(k+1).

In the position shift/resolution conversion processing TR_(k+1), byusing the position aligned data PR_(k+1) generated by the paralleldisplacement processing PM_(k+1) shown in FIG. 4, the processing circuit8 aligns the position of the super resolution estimation image dataZ_(n, k) with the position of the low resolution image data L_(k+1), andthereafter the resolution is converted into the low resolution. Theposition shift/resolution conversion processing TR_(k+1) is indicated bythe matrix W_(k+1) in the formula (14).

Next, in the difference processing SUB_(k+1), the processing circuit 8generates difference image data “W_(k+1)Z_(n, k)−L_(k+)” representativeof a difference between the low resolution estimation image datagenerated by the position shift/resolution conversion processingTR_(k+1) and the low resolution image data L_(k+).

Next, in a gradient image generation processing D_(k+1), the processingcircuit 8 generates the matrices M_(k+1), M_(k+1) ^(T) by using themoving image data MI_(k+1) generated by the processing shown in FIG. 4.

The matrix M_(k+1) does not change the value of an object withoutmotion, but lowers the value of an object with motion. Since the lowresolution image data L_(k+1) is used as a reference pixel position inthe matrix MI, the pixel position of the moving image data MI_(k+1) iscorrected.

In the gradient image generation processing D_(k+1), the processingcircuit 8 converts the difference image data generated by the differenceprocessing SUB_(k+1) by using the matrix W_(k+1) ^(T)M_(k+) ^(T)M_(k+1)to generate “W_(k+1)z_(n, k)−L_(k+1))”.

In the gradient image generation processing D_(k+1), the processingcircuit 8 adds “W_(k−1)M_(k−1)^(T)M_(k−1)(W_(k−1)Z_(n, k)−L_(k−1))+W_(k) ^(T)(W_(k)Z_(n, k)−L_(k))”generated by the gradient image generation/update processing D_(k−1) and“W_(k+1)M_(k+) ^(T)M_(k+1)(W_(k+1)Z_(n, k)−L_(k+1))”, thereby generatingthe second term in the formula (14).

Position Shift Processing TR1, Difference Processing SUB1

In the position shift processing TR1, the processing circuit 8 executesa position alignment processing by motion compensation for the superresolution estimation image data Z_(n, k) (matrix W_(zk)).

In the difference processing SUB1, the processing circuit 8 generatesdifference data “W_(Z k)Z_(n, k)−Z_(k−1)” representative of a differencebetween the super resolution estimation image data Z_(n, k) after theposition alignment and the super resolution image data Z_(k−1) generatedby the super resolution conversion processing SP_(k−1) shown in FIG. 2.

Gradient Image Update Processing D1

In a gradient image update processing D1, the processing circuit 8converts (W_(zk)Z_(n, k)−Z_(k−1)) generated by the difference processingSUB1 by using a conversion 2W_(zk) ^(T)M_(zk) ^(T)M_(zk).

The matrix M_(k−1) used by the gradient image generation processingD_(k−1) is used as the conversion matrix M_(zk).

The matrix W_(k−1) used by the position shift/resolution conversionprocessing TR_(k−1) is used as the conversion matrix W_(zk).

The conversion result corresponds to the processing result of the firstterm in the formula (14).

In the gradient image update processing D1, the processing circuit 8adds the processing result of the second term in the formula (14)generated by the gradient image generation/update processing D_(k+1) andthe processing result of the first term in the formula (14) generated bythe conversion.

Image Update Processing D2

In the image update processing D2, the processing circuit 8 executes ahigh pass filter processing C for the super resolution estimation imagedata Z_(n, k) and adds the filter output to the addition result by thegradient image update processing D1 to generate Δf(z_(n, k)) in theformula (14).

The processing circuit 8 multiplies Δf(Z_(n, k)) by a predeterminedscalar value α to generate new super resolution estimation image dataZ_(n+1, k).

The processing circuit 8 judges whether Δf(z_(n, k)) is equal to orsmaller than a predetermined threshold value.

If the processing circuit 8 judges that Δf(z_(n, k)) is equal to orsmaller than the predetermined threshold value, the super resolutionestimation image data Z_(n, k) is used as the super resolutionestimation image data Z_(k).

If the processing circuit 8 judges that Δf(Z_(n, k)) is neither equal tonor smaller than the predetermined threshold value, the processing shownin FIG. 6 are repeated for the new super resolution estimation imagedata Z_(n+1, k).

Example of an Overall Operation of the Image Processing Apparatus 1

FIGS. 7 and 8 are flow charts illustrating an example of the overalloperation of the image processing apparatus 1.

Steps ST1 and ST2:

The processing circuit 8 shown in FIG. 1 executes the processing shownin FIG. 4 to generate position alignment data PR_(k) and moving imagedata MI_(k) for the low resolution image data L_(k) and L_(k+1).

Step ST3:

By using the generated position alignment data PR_(k), the lowresolution image data Z_(k) and the super resolution image data Z_(k−1),the processing circuit 8 executes the super resolution imageinitializing processing described with reference to FIG. 5 to generatesuper resolution estimation initial image data Z_(0, k).

Step ST4:

The processing circuit 8 initializes the number n of repetition times to0 and sets constants a1 and a2 for designating low resolution images tobe input for a super resolution processing.

As described earlier, a low resolution image at a time k and a lowresolution images before and after the time k are used (“a” is a presetconstant). Since an exceptional processing is required for processingfirst “a” frames and last “a” frames of moving images, a1 and a2 areset.

The processing circuit 8 sets a1=k if k<a, and a1=a if k≧a. Theprocessing circuit 8 sets a2=kmax−k if k+a>kmax, and a2=a if k+a≦kmax.

Step ST5:

The processing circuit 8 initializes Δf(Z_(n, k)) to a null vector.

Step ST6:

The processing circuit 8 substitutes “k−a1” into 1.

Step ST7:

The processing circuit 8 executes the position shift/resolutionconversion processing W_(k−1), Wk or W_(k+1) shown in FIG. 6.

Step ST8:

The processing circuit 8 executes the difference processing SUB_(k−1),SUB_(k) or SUB_(k+1) and the gradient image generation processingD_(k−1), D_(k) or D_(k+1).

Step ST9:

The processing circuit 8 increments the variable 1 by “1”.

Step ST10:

The processing circuit 8 judges whether the variable 1 is larger than“k+2a”, and if it is judged larger, the flow advances to Step ST11,whereas if not, the flow returns to Step ST7.

Step ST11:

The processing circuit 8 judges whether k is “0”, and if k is “0”, theflow skips to Step ST13, whereas if not, the flow advances to Step ST12.

Step ST12:

The processing circuit 8 executes the position shift processing TR1 anddifference processing SUB1 shown in FIG. 6.

Step ST13:

The processing circuit 8 executes the gradient image update processingD1 shown in FIG. 6 to add the calculation result of the second term inthe formula (14) generated by the gradient image generation/updateprocessing D_(k+1) and the processing result of the first term in theformula (14) generated by the conversion.

Step ST14:

The processing circuit 8 executes the image update processing D2 shownin FIG. 6 to generate new super resolution estimation image dataZ_(n+1, k).

Step ST15:

The processing circuit 8 increments the variable n by “1”.

Step ST16:

The processing circuit 8 judges whether the gradient image Δf generatedby the image update processing D is smaller than “fmax”, and if it isjudged smaller, the processing is terminated and the super resolutionestimation image data Z_(n, k) is used as the super resolution imagedata Z_(k).

If the processing circuit 8 judges that the gradient image Δf generatedby the image update processing D is not smaller than “fmax”, the flowadvances to Step ST17.

Step ST17:

The processing circuit 8 judges whether the variable n is equal to orlarger than “nmax”, and if it is judged that the variable is equal to orlarger than “nmax”, the processing is terminated (the super resolutionestimation image data Z_(n, k) is used as the super resolution imagedata Z_(k)), whereas if not, the flow returns to Step ST5.

As described so far, according to the image processing apparatus 1, theposition of the super resolution image data Z_(k−1) of the alreadygenerated low resolution image data L_(k−1) is aligned with the positionof the low resolution image data L_(k), and the super resolution imagedata Z_(k) is sequentially updated in the manner to make small adifference between the super resolution image data and low resolutionimage data.

The super resolution image data can be generated by a smallercalculation amount than that used by a related art's system whichcalculates motion vector for each pixel data.

In a related art system, motion vector may not be calculated properlybecause of superposition of objects, aliasing or noises. According tothe image processing apparatus 1, for the motion compensation, after themotion compensation for a whole image is performed, presence/absence ofa motion of each individual object other then a motion of the wholeimage is detected, and in accordance with this detection, the superresolution processing is performed for an area of an object withoutmotion. It is therefore possible to suppress the influence of aliasingand noises. An area of an object with motion is not reflected upon thesuper resolution processing result, so that object blurring is hard tobe formed.

Since the image processing apparatus 1 processes a plurality ofconsecutive images of moving images, it is possible to suppress noiseschanging with time, flicker of a fluorescent lamp and the like.

A weight of a moving object area indicated by the formula (14) becomessmall because of motion compensation for a motion of an individualobject. With this small weight, an update amount for other still objectareas becomes small. Namely, a moving object area of the superresolution image at the time k is an image similar to a simple enlargedimage of the low resolution image data L_(k) at the time k. Since theimage processing apparatus 1 considers the following two points, properprocessing can be executed.

(1) It is assumed that the low resolution image data L_(k−1) and L_(k+1)at the times k−1 and k+1 and also the super resolution image dataZ_(k−1) at the time k−1 largely contribute also to the moving objectarea. With this assumption, a moving object in the resultant image Z_(k)has an image tailing the object and becomes an unnatural image.

(2) When a moving image is reproduced actually, human being cannot sensea fine image at a high resolution in the moving object area.

The present invention is not limited to the above-described embodiment.Namely, it will be apparent to those skilled in the art that variousmodifications, combinations, sub-combinations and substitutes can bemade without departing from the technical range of the present inventionor in the equivalent range thereof.

For example, although the processing for moving images of a grey scalehas been described in the embodiments, the present invention is alsoapplicable to color moving images. A color image is separated intoluminance components and color components, and motion compensationincluding position alignment and the like is performed for the luminancecomponents. By using this motion compensation result, the superresolution processing with the iterative processing is executed for theluminance components and color components. The super resolutionprocessing may be executed by synthesizing image data generated by thesuper resolution processing with the iterative processing and executedonly for the luminance components and image data generated by enlargingthe color components by the bicubic interpolation or the like. In thiscase, a calculation cost can be reduced.

Although all three terms in the right side of the formula (14) are usedto generate Δf in the embodiments, Δf may be generated by using only thefirst term, second term or both first and second terms.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

1. An image processing method for converting moving image data formedwith a plurality of first image data having a first resolution intomoving image data formed with a plurality of second image data having asecond resolution, the plurality of first image data being sequentiallyprocessed for display, the image processing method comprising: a firststep including generating position alignment data between the firstimage data to be processed and another first image data other than thefirst image data to be processed; a second step including aligning aposition of candidate image data having the second resolutioncorresponding to the first image data to be processed with a position ofthe other first image data, in accordance with the position alignmentdata generated by the first step; a third step including updating thecandidate image data so as to minimize a first difference between theother first image data and the candidate image data position-aligned bythe second step; and a fourth step including executing the second stepand the third step at least one time, and using the candidate image datahaving the difference equal to or smaller than a reference value, as thesecond image data corresponding to the first image data to be processed.2. The image processing method according to claim 1, further comprising:a fifth step including detecting moving image data in the first imagedata to be processed, in accordance with the first image data to beprocessed and the other first image data; wherein the third stepincludes generating difference data between the other first image dataand the candidate image data position-aligned by the second step, as thefirst difference, and includes executing a processing for reducing avalue of the difference data corresponding to a moving area, inaccordance with the moving image data generated by the fifth step, andupdates the candidate image data so as to reduce the difference dataafter subjected to such processing.
 3. The image processing methodaccording to claim 1, wherein the first step including generating theposition alignment data between the first image data to be processed andthe other image data, the position alignment data being representativeof a position alignment amount relative to a whole image.
 4. The imageprocessing method according to claim 1, further comprising: a sixth stepincluding generating an initial value of the candidate image data inaccordance with the first image data to be processed and the secondimage data already generated for another first image data other than thefirst image data to be processed; wherein the initial value generated bythe sixth step is used as the candidate image data at the second step tobe executed before the third step is executed for the first time.
 5. Theimage processing method according to claim 1, wherein the third stepincludes aligning a position of the candidate image data with a positionof the second image data already generated for another first image dataother than the first image data to be processed, and after positionalignment, generates a second difference therebetween, and updates thecandidate image data so as to further reduce the second difference. 6.The image processing method according to claim 1, wherein the third stepincludes updating the candidate image data so as to further reduce atotal sum of data obtained by subjecting the candidate image data to ahigh pass filter processing, the first difference and the seconddifference.
 7. An image processing apparatus for converting moving imagedata formed with a plurality of first image data having a firstresolution into moving image data formed with a plurality of secondimage data having a second resolution, the plurality of first image databeing sequentially processed for display, the image processing apparatuscomprising: first means for generating position alignment data betweenthe first image data to be processed and another first image data otherthan the first image data to be processed; second means for aligning aposition of candidate image data having the second resolutioncorresponding to the first image data to be processed with a position ofthe other first image data, in accordance with the position alignmentdata generated by the first means; third means for updating thecandidate image data so as to minimize a first difference between theother first image data and the candidate image data position-aligned bythe second means; and fourth means for executing processing by thesecond means and the third means at least one time, and using thecandidate image data having the difference equal to or smaller than areference value, as the second image data corresponding to the firstimage data to be processed.
 8. A program to be executed by an imageprocessing apparatus for converting moving image data formed with aplurality of first image data having a first resolution into movingimage data formed with a plurality of second image data having a secondresolution, the plurality of first image data being sequentiallyprocessed for display, the program comprising: a first procedure forgenerating position alignment data between the first image data to beprocessed and another first image data other than the first image datato be processed; a second procedure for aligning a position of candidateimage data having the second resolution corresponding to the first imagedata to be processed with a position of the other first image data, inaccordance with the position alignment data generated by the firstprocedure; a third procedure for updating the candidate image data so asto minimize a first difference between the other first image data andthe candidate image data position-aligned by the second procedure; and afourth procedure for executing the second procedure and the thirdprocedure at least one time, and using the candidate image data havingthe difference equal to or smaller than a reference value, as the secondimage data corresponding to the first image data to be processed.
 9. Arecording medium storing a program to be executed by an image processingapparatus for converting moving image data formed with a plurality offirst image data having a first resolution into moving image data formedwith a plurality of second image data having a second resolution, theplurality of first image data being sequentially processed for display,the program comprising: a first procedure for generating positionalignment data between the first image data to be processed and anotherfirst image data other than the first image data to be processed; asecond procedure for aligning a position of candidate image data havingthe second resolution corresponding to the first image data to beprocessed with a position of the other first image data, in accordancewith the position alignment data generated by the first procedure; athird procedure for updating the candidate image data so as to minimizea first difference between the other first image data and the candidateimage data position-aligned by the second procedure; and a fourthprocedure for executing the second procedure and the third procedure atleast one time, and using the candidate image data having the differenceequal to or smaller than a reference value, as the second image datacorresponding to the first image data to be processed.
 10. An imageprocessing method for converting moving image data formed with aplurality of first image data having a first resolution into movingimage data formed with a plurality of second image data having a secondresolution, the plurality of first image data being sequentiallyprocessed for display, the image processing method comprising:generating position alignment data between the first image data to beprocessed and another first image data other than the first image datato be processed; aligning a position of candidate image data having thesecond resolution corresponding to the first image data to be processedwith a position of the other first image data, in accordance with theposition alignment data generated; updating the candidate image data soas to minimize a first difference between the other first image data andthe candidate image data position-aligned; and executing the aligningand the updating at least one time, and using the candidate image datahaving the difference equal to or smaller than a reference value, as thesecond image data corresponding to the first image data to be processed.11. An image processing apparatus for converting moving image dataformed with a plurality of first image data having a first resolutioninto moving image data formed with a plurality of second image datahaving a second resolution, the plurality of first image data beingsequentially processed for display, the image processing apparatuscomprising: a processing circuit; and a memory; wherein the processingcircuit generates position alignment data between the first image datato be processed and another first image data other than the first imagedata to be processed; aligns a position of candidate image data havingthe second resolution corresponding to the first image data to beprocessed with a position of the other first image data, in accordancewith the position alignment data generated; updates the candidate imagedata so as to minimize a first difference between the other first imagedata and the candidate image data position-aligned; and executes thealigning and the updating at least one time, and using the candidateimage data having the difference equal to or smaller than a referencevalue, as the second image data corresponding to the first image data tobe processed.